Method of generating log data, non-transitory computer-readable medium, and data structure

ABSTRACT

A method of generating log data that includes a plurality of records, in which information detected through image recognition processing on a plurality of image frames is recorded. The method includes, detecting objects in each image frame through the image recognition processing, generating a record including identification information of each image frame and the aggregated number of objects detected in the image frame for each image frame, and adding metadata relating to the detected objects to the corresponding record when the aggregated number of objects is equal to or greater than one. When the aggregated number of objects is zero, the metadata related to the detected objects is not added to the record.

INCORPORATION BY REFERENCE

The disclosure of Japanese Patent Application No. 2018-166253 filed onSep. 5, 2018 including the specification, drawings and abstract isincorporated herein by reference in its entirety.

BACKGROUND 1. Technical Field

The disclosure relates to a method of generating log data, anon-transitory computer-readable medium storing a program that causesprocessing to be executed, and a data structure.

2. Description of Related Art

In the related art, a technique that increases a speed of search forinformation included in data is known. For example, Japanese Patent No.3012677 discloses a technique that rearranges a search sequence of thesubsequences according to the number of searches regarding registeredsubsequences in dictionary data in order to increase a speed of searchof dictionary data to be used in encoding.

SUMMARY

There is a need to increase a speed of search processing of log data ofimage recognition processing on video captured by an in-vehicle cameraor the like. In general, a long time is taken for the search processingdue to the expansion of a data size or the like of the log data of theimage recognition processing. For this reason, there is room forimprovement on an information search technique of the log data of theimage recognition processing.

The disclosure provides a method of generating log data, anon-transitory computer-readable medium storing a program that causesprocessing to be executed, and a data structure for improving aninformation search technique of log data of image recognitionprocessing.

A first aspect of the disclosure relates to a method of generating logdata including a plurality of records, in which information detectedthrough image recognition processing on a plurality of image frames isrecorded. The method includes detecting objects in each image framethrough the image recognition processing, generating, for each imageframe, a record including identification information of a correspondingimage frames and the aggregated number of objects detected in thecorresponding image frames, and adding metadata related to the detectedobjects to a corresponding record when the aggregated number of objectsis equal to or greater than one. When the aggregated number of objectsis zero, the metadata related to the detected objects is not added tothe record.

The method according to the first aspect may further include adding thetotal number of objects for each object classification to thecorresponding record when the aggregated number of objects is equal toor greater than one.

In the method according to the first aspect, a sequence of metadata inthe log data may be determined based on at least one of a recognitionfrequency and a detection score of each object.

The method according to the first aspect may further include changing asequence of metadata in the log data in a predetermined period based onat least one of a recognition frequency and a detection score of eachobject in the predetermined period, and recording information of thesequence in a sequence table.

The method according to the first aspect may further include measuring asearch speed of the log data with a predetermined search query based ontag information included in the log data, and changing the taginformation when the search speed is lower than a predetermined value.

In the method according to the first aspect, at least a part of objectclassifications may be hierarchized. Solely when a detection score ofimage recognition processing related to a higher hierarchical objectclassification is equal to or greater than a predetermined value, imagerecognition processing related to a lower hierarchical objectclassification of the higher hierarchical object classifications may beexecuted.

A second aspect of the disclosure relates to a non-transitorycomputer-readable medium storing a program that causes a computerfunctioning as an information processing device configured to generatelog data including a plurality of records, in which information detectedthrough image recognition processing on a plurality of image frames isrecorded, to execute processing. The processing includes detectingobjects in each image frame through the image recognition processing,generating, for each image frame, a record including identificationinformation of a corresponding image frame and the aggregated number ofobjects detected in the corresponding image frame, and adding metadatarelated to the detected objects to a corresponding record when theaggregated number of objects is equal to or greater than one. When theaggregated number of objects is zero, the metadata related to thedetected objects is not added to the record.

A third aspect of the disclosure relates to a data structure of log dataincluding information detected through image recognition processing on aplurality of image frames. The data structure includes a plurality ofrecords corresponding to the image frames. Each record includesidentification information of a corresponding image frame and theaggregated number of objects detected in the corresponding image frames.A record in which the aggregated number of objects is equal to orgreater than one includes metadata related to the detected objects, anda record in which the aggregated number of objects is zero includes nometadata.

A fourth aspect of the disclosure relates to a data structure of logdata that is used in an information processing system including avehicle and a server, and includes information detected through imagerecognition processing on a plurality of image frames. The datastructure includes a plurality of records corresponding to the imageframes. Each record includes identification information of acorresponding image frames and the aggregated number of objects detectedin the corresponding image frames. The vehicle adds metadata related tothe detected objects to the corresponding record when the aggregatednumber of objects is equal to or greater than one and adds no metadatawhen the aggregated number of objects is zero.

With the method of generating log data, the non-transitorycomputer-readable medium storing a program that causes processing to beexecuted, and the data structure according to the aspects of thedisclosure, it is possible to improve an information search technique oflog data of image recognition processing.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, advantages, and technical and industrial significance ofexemplary embodiments of the disclosure will be described below withreference to the accompanying drawings, in which like numerals denotelike elements, and wherein:

FIG. 1 is a diagram showing the schematic configuration of aninformation processing system according to an embodiment of thedisclosure;

FIG. 2 is a block diagram showing the schematic configuration of avehicle;

FIG. 3 is a block diagram showing the schematic configuration of aserver;

FIG. 4 is a diagram showing a data structure of log data;

FIG. 5 is a conceptual diagram showing a mode of image recognitionprocessing;

FIG. 6 is a diagram showing a data structure of a data area of log data;

FIG. 7 is a diagram showing a data structure of log data in a case wherethe aggregated number of objects is zero;

FIG. 8 shows an example of generated log data;

FIG. 9 is a flowchart showing a generation operation of log data in theinformation processing system according to the embodiment of thedisclosure;

FIG. 10 shows an example of a sequence table; and

FIG. 11 is a conceptual diagram representing the hierarchicalrelationship between objects to be detected.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the disclosure will be described.

Configuration of Information Processing System

The outline of an information processing system 1 according to anembodiment of the disclosure will be described referring to FIG. 1. Theinformation processing system 1 includes a vehicle 10 and a server 20.The vehicle 10 is, for example, an automobile; however, the vehicle 10is not limited to an automobile and may be any vehicle. In FIG. 1, forsimplification of description, solely one vehicle 10 is shown; however,the number of vehicles 10 in the information processing system 1 may beone or more. The server 20 includes one server device or a plurality ofserver devices that can perform communication with one another. Theserver 20 can perform communication with a client terminal through anetwork 30. In the embodiment, a client includes one or more vehicles10. Accordingly, each vehicle 10 and the server 20 can performcommunication through the network 30 including, for example, a mobilecommunication network, the Internet, and the like. However, the clientthat can perform communication with the server 20 is not limited to thevehicle 10, and may include, for example, any information processingdevice, such as a smartphone or a computer.

As the outline of the embodiment, image recognition processing isexecuted on a plurality of captured images (hereinafter, referred to as“image frames”) obtained by the vehicle 10 imaging scenery outside thevehicle to generate log data. Each record of the log data includes theaggregated (hereinafter, referred to as “the aggregated number ofobjects”) of objects detected through the image recognition processing.In a case where the aggregated number of objects is equal to or greaterthan one, detailed information (hereinafter, referred to as “metadata”)related to each object is added to the record. In a case where theaggregated number of objects is zero, metadata is not added to therecord.

In this way, according to the embodiment, in a case where the aggregatednumber of objects is equal to or greater than one, the metadata of thedetected object is added to the log data, and in a case where theaggregated number of objects is zero, unneeded data is not added to thelog data, whereby it is possible to suppress an increase in data size ofthe log data. An increase in data size of the log data to be searched issuppressed, whereby it is possible to improve a search speed and toimprove an information search technique of the log data.

Configuration of Vehicle

As shown in FIG. 2, the vehicle 10 includes a communication device 11,an information processing device 12, and an imaging device 13. Thecommunication device 11 and the information processing device 12 areconnected to perform communication with each other through an in-vehiclenetwork, such as a controller area network (CAN), or a dedicated line.

The communication device 11 is, for example, in-vehicle communicationequipment, such as a data communication module (DCM). Specifically, thecommunication device 11 includes a communication unit 111, a storageunit 112, and a controller 113.

The communication unit 111 includes a communication module that performscommunication through the in-vehicle network or the dedicated line. Thecommunication unit 111 includes a communication module that is connectedto the network 30. For example, the communication unit 111 may include acommunication module corresponding to a mobile communication standard,such as 4th Generation (4G) and 5th Generation (5G). In the embodiment,the vehicle 10 is connected to the network 30 through the communicationunit 111.

The storage unit 112 includes one or more memories. In the embodiment,the “memory” is, for example, a semiconductor memory, a magnetic memory,an optical memory, or the like, but is not limited thereto. Each memoryincluded in the storage unit 112 may function as, for example, a mainstorage device, an auxiliary storage device, or a cache memory. Thestorage unit 112 stores optional information that is used for theoperation of the communication device 11. For example, the storage unit112 may store a system program, an application program, identificationinformation of the vehicle 10, and the like. The identificationinformation of the vehicle 10 is information capable of uniquelyidentifying the vehicle 10 in the information processing system 1. Wheninformation is transmitted from the communication device 11 to theserver 20, the identification information of the vehicle 10 istransmitted to the server 20 along with the above-described information,whereby the server 20 can identify the vehicle 10 as a transmissionsource. Identification information of the communication device 11 or theinformation processing device 12 in the vehicle 10 may be used as theidentification information of the vehicle 10. Information stored in thestorage unit 112 may be updated with, for example, information to beacquired from the network 30 through the communication unit 111.

The controller 113 includes one or more processors. In the embodiment,the “processor” is a general-purpose processor or a dedicated processorspecific to specified processing, but is not limited thereto. Thecontroller 113 controls the overall operation of the communicationdevice 11. In the embodiment, the vehicle 10 and the server 20 performcommunication through the communication device 11 that is controlled bythe controller 113.

For example, the controller 113 acquires the log data related to theimage frames obtained by imaging scenery outside the vehicle from theinformation processing device 12. The controller 113 transmits theacquired log data to the server 20. The log data may be transmitted eachtime each record is generated or may be transmitted after beingaccumulated in the storage unit 112 in a given period. The controller113 may acquire the image frames from the imaging device 13 and maytransmit the image frames to the server 20.

The information processing device 12 is, for example, a navigationdevice or an autonomous driving control device that is mounted in thevehicle 10, but is not limited thereto. Specifically, the informationprocessing device 12 includes a communication unit 121, a storage unit122, a positional information acquisition unit 123, an output unit 124,an input unit 125, and a controller 126.

The communication unit 121 includes a communication module that performscommunication through the in-vehicle network or the dedicated line.

The storage unit 122 includes one or more memories. Each memory includedin the storage unit 122 may function as, for example, a main storagedevice, an auxiliary storage device, or a cache memory. The storage unit122 stores optional information that is used for the operation of theinformation processing device 12. For example, the storage unit 122 maystore a system program, an application program, and the like.Information stored in the storage unit 122 may be updated with, forexample, information to be acquired from the network 30 through thecommunication device 11.

The positional information acquisition unit 123 includes one or morereceivers corresponding to any satellite positioning system. Forexample, the positional information acquisition unit 123 may include aglobal positioning system (GPS) receiver. The positional informationacquisition unit 123 acquires positional information of the vehicle 10and outputs the acquired positional information to the controller 126.

The output unit 124 includes one or more output interfaces that outputinformation to notify a user of information. For example, the outputinterfaces included in the output unit 124 are a display that outputsinformation in a form of video, a speaker that outputs information in aform of sound, and the like, but are not limited thereto. For example,the display is a panel display, a head-up display, or the like, but isnot limited thereto. In the embodiment, “video” may include text, astill image, and a moving image.

The input unit 125 includes one or more input interfaces that detectuser input. For example, the input interfaces included in the input unit125 is physical keys, capacitance type keys, a touch screen providedintegrally with the panel display of the output unit 124, a microphonethat receives sound input, and the like, but are not limited thereto.

The controller 126 includes one or more processors. The controller 126controls the overall operation of the information processing device 12.For example, the controller 126 generates log data based on thepositional information of the vehicle 10 acquired from the positionalinformation acquisition unit 123 and the image frames acquired from theimaging device 13.

The imaging device 13 is a device that generates a plurality of imageframes obtained by imaging a subject. In the embodiment, the imagingdevice 13 is provided in the vehicle 10 such that scenery outside thevehicle can be imaged. The imaging device 13 may be, for example, anin-vehicle camera that is used for driving assistance of the vehicle, adrive recorder, or the like. Specifically, the imaging device 13includes a communication unit 131, a storage unit 132, an imaging unit133, and a controller 134.

The communication unit 131 includes a communication module that performscommunication through the in-vehicle network or the dedicated line.

The storage unit 132 includes one or more memories. Each memory includedin the storage unit 132 may function as, for example, a main storagedevice, an auxiliary storage device, or a cache memory. The storage unit132 stores optional information that is used for the operation of theimaging device 13. For example, the storage unit 132 may store a systemprogram, an application program, and the like. Information stored in thestorage unit 132 may be updated with, for example, information to beacquired from the network 30 through the communication device 11.

The imaging unit 133 includes an optical element, such as a lens, and animage sensor. In the embodiment, the “image sensor” is, for example, acomplementary metal-oxide-semiconductor (CMOS) image sensor, acharge-coupled devices (CCD) image sensor, or the like, but is notlimited thereto.

The controller 134 includes one or more processors. The controller 134controls the overall operation of the imaging device 13.

For example, the controller 134 generates a plurality of image framesobtained by imaging scenery outside the vehicle 10 with the imaging unit133 at a predetermined frame rate (for example, 60 fps), and outputs theimage frames to the information processing device 12. The image framesmay be output to the information processing device 12 each time eachimage frame is generated or may be output to the information processingdevice 12 after being accumulated in the storage unit 132 in a givenperiod. The image frames may be output to the information processingdevice 12 as a moving image including the image frames. In other words,the controller 134 may output the image frames as a plurality of stillimage files (for example, JPG files) or may output the image frames asone moving image file (for example, an AVI file).

Configuration of Server

As shown in FIG. 3, the server 20 includes a server communication unit21, a server storage unit 22, and a server controller 23.

The server communication unit 21 includes a communication module that isconnected to the network 30. For example, the server communication unit21 may include a communication module corresponding to a wireless localarea network (LAN) standard. In the embodiment, the server 20 isconnected to the network 30 through the server communication unit 21.

The server storage unit 22 includes one or more memories. Each memoryincluded in the server storage unit 22 may function as, for example, amain storage device, an auxiliary storage device, or a cache memory. Theserver storage unit 22 stores optional information that is used for theoperation of the server 20. For example, the server storage unit 22 maystore a system program, an application program, a database that storesthe log data received from the vehicle 10, and the like. Informationstored in the server storage unit 22 may be updated with, for example,information to be acquired from the network 30 through the servercommunication unit 21.

The server controller 23 shown in FIG. 3 includes one or moreprocessors. The server controller 23 controls the overall operation ofthe server 20. For example, the server controller 23 receives andacquires the log data from the vehicle 10. The server controller 23performs search processing of the log data based on a search query andoutputs a search processing result. The search query is transmitted fromthe client terminal requesting the reference of the log data to theserver 20 through the network 30. The search query includes informationof any search condition designated by a keyword, a tag, and the like.Hereinafter, the structure of the log data for improving the searchprocessing will be described.

Structure of Log Data

As described above, the log data is generated based on the image framesreceived from the imaging device 13 by the controller 126 of theinformation processing device 12 in the vehicle 10. The log dataincludes a plurality of records corresponding to the image frames.Specifically, the controller 126 detects objects in each image framethrough the image recognition processing on each image frame. Thecontroller 126 counts the aggregated number of objects. For each imageframe, the controller 126 generates a record including identificationinformation of the image frame and the aggregated number of objectsdetected in the image frame. In a case where the aggregated number ofobjects is equal to or greater than one, the controller 126 adds thetotal number (hereinafter, referred to as “the number of detections”) ofeach object per object classification and the metadata related to eachobject to the corresponding record.

FIG. 4 shows the data structure of the log data. As shown in FIG. 4,each record of the log data includes a header area and a data area. Theheader area includes frame identification information, a time stamp, theaggregated number of objects, each number of detections, and taginformation. The data area includes metadata. The frame identificationinformation is information for uniquely specifying an image frame in thesystem. The time stamp is date and time information on which the imageframe is generated.

In an example of FIG. 4, the classifications of the objects to bedetected are 26 types of a to z. The number of detections (a) to thenumber of detections (z) in FIG. 4 represent the number of detections ofthe objects of the object classifications a to z, respectively. Thenumber of classifications of the objects to be detected is not limitedto 26, and may be less than 26 or may be greater than 26. The taginformation is optional information related to the image frame, andincludes, for example, angle information, pedestrian and the likecrossing information, and brightness information. The angle informationindicates an angle between a lane and the vehicle 10 detected throughthe image recognition processing. The pedestrian and the like crossinginformation is information indicating that a pedestrian, a bicycle, orthe like is detected on a traveling lane. The brightness information isinformation related to brightness in front of the vehicle to bedetermined from the image frame. The tag information may include thepositional information of the vehicle 10 output from the positionalinformation acquisition unit 123. The tag information is added to therecord corresponding to the image frame, whereby it is possible toperform a conditional search of the log data with a tag.

The data area of each record includes metadata of each of all objectsdetected through the image recognition processing. The metadata includesa detection frame coordinate (x), a detection frame coordinate (y), adetection frame width, a detection frame height, an objectclassification, an absolute position, a relative position, a distance, adetection score, a detection count, a degree of risk, and a signalcolor. The detection frame coordinate (x) and the detection framecoordinate (y) are reference coordinates of a rectangular detectionframe including the detected object. FIG. 5 shows a detection frame in acase where an object is detected in a captured image frame. In FIG. 5,an automobile is detected. For example, an x coordinate and a ycoordinate of an upper left vertex P of the detection frame shown inFIG. 5 may be the detection frame coordinate (x) and the detection framecoordinate (y), respectively. A position and a size of the detectionframe are specified by the coordinates of the vertex P, a detectionframe width W, and a detection frame height H.

The object classification indicates the type of an object, and Table 1shows an example of the object classification.

TABLE 1 Object Classification ID Object Classification  0 Pedestrian  1Bicycle  2 Motorcycle  3 Automobile . . . . . .  7 Traffic Sign . . . .. . 10 Falling Object 11 Road Cone 12 Construction Site 13 TrafficSignal 14 Traffic Signal (Arrow) 15 Signboard (Sign) 16 Signboard(Price) 17 Face (for Privacy Mask) 18 Number Plate (for Privacy Mask) .. . . . .

The absolute position and the relative position are informationindicating a position of an object based on a traveling lane of thevehicle. The absolute position specifies respective lanes in a roadhaving six lanes in both directions by values of 1 to 6 from a travelinglane at a left end to a traveling lane at a right end. The relativeposition sets the traveling lane, on which the vehicle 10 is present, tozero, represents a traveling lane on a right side by a positive number,and represents a traveling lane of a left side by a negative number.

The distance represents a direct distance to the detected object. Thedetection score is a value indicating a degree of confidence of theimage recognition processing. When the detection score is higher, adetection result of the image recognition processing is probable. Thedetection count is the number of detections when the same target isdetected in a plurality of coordinates.

The degree of risk is determined based on, for example, the followingcriterion.

(i) In a case where another vehicle is detected on the traveling lane ofthe host vehicle, the degree of risk is determined in four stages (0 to3) based on a relative speed (at the time of approach) and aninter-vehicle distance.

(ii) In a case where a pedestrian or a bicycle is detected on anytraveling lane, the degree of risk is determined in four stages (0 to 3)based on the distance between the pedestrian or the bicycle and the hostvehicle.

(iii) Otherwise, the degree of risk is set to a predetermined value (forexample, “−1”).

The signal color indicates a detected lamp color of a traffic signalwith a numerical value, and is determined based on, for example, Table2. In a case where the detected object is other than a traffic signal,the signal color is set to a predetermined value (for example, “−1”).

TABLE 2 Lamp Color Value Green 0 Yellow 1 Red 2 Unclear 3

FIG. 6 is a diagram showing the structure of the data area of the logdata. As shown in FIG. 6, metadata related to all detected objects isstored in the data area. The data area is an area having a variablelength, and the metadata for the aggregated number of objects is storedin the data area.

The metadata is arranged following the values and the sequence of thenumber of detections (a) to the number of detections (z) included in theheader area of the record. In other words, the structure of the dataarea is determined based on each number of detections and the sequenceof each number of detections included in the header area. For example,in a case where the objects of the object classifications a, b, c aredetected by i, j, and k pieces, respectively, as shown in FIG. 6, ipieces of metadata of the object classification a are continued, andsubsequently, j pieces of metadata of the object classification b arecontinued. In addition, k pieces of metadata of the objectclassification c are continued subsequently.

As a comparative example of the data structure of the data area, a casewhere the data area is set to a fixed length and a storage area ofmetadata of each object classification is sufficiently secured inadvance is considered. In a case where the structure of the comparativeexample is employed, the structure of the data area can be formed in asimple format; however, an area for storing metadata should besufficiently secured. Furthermore, since data is set to a fixed length,a default value (for example, 0) needs to be stored in the storage areaof the metadata. For this reason, the data size is the same regardlessof the number of objects to be detected, and wasteful data is included.According to the embodiment, since the data area of the log data is setto a variable length, and solely the metadata for the number of objectsto be detected is included, it is possible to suppress an increase indata size of the log data. With the use of information of each number ofdetections, it is possible to easily specify an object classificationand a portion of the data area of the log data where the metadata of theobject classification is present.

The sequence of each number of detections of the object classificationsa to z is determined based on, for example, at least one of arecognition frequency and a detection score of each object. With this,the sequence of the metadata to be stored in the data area isdetermined. That is, the sequence of the metadata is determined based onat least one of the recognition frequency and the detection score ofeach object. For example, the metadata may be arranged in order from anobject classification for which an average value of a recognitionfrequency of an object is high. In this case, the metadata of theobjects for which the average value of the recognition frequency is highis present ahead (in FIG. 6, the left side). In this way, in a casewhere scanning is performed from the left side of the record in thesearch processing of the log data, it is possible to improve a searchspeed for an object for which a recognition frequency is high.

For example, the metadata may be arranged in order from an objectclassification for which an average value of a detection score of anobject is high. In this case, the metadata of the object for which theaverage value of the detection score is high is present ahead (in FIG.6, the left side). In this way, in a case where scanning is performedfrom the left side of the record in the search processing of the logdata, it is possible to improve a search speed for an object for which adetection score is high.

The sequence of the metadata may be determined based on a searchfrequency of an object, and for example, the metadata may be arranged inorder from an object for which a search frequency is high. In this way,in a case where scanning is performed from the left side of the recordin the search processing of the log data, it is possible to improve asearch speed for an object for which a search frequency is high.

The controller 126 does not add the total number (the number ofdetections) of the objects for each object classification and themetadata to the log data in a case where the aggregated number ofobjects is zero. FIG. 7 shows a data structure of log data in a casewhere the aggregated number of objects is zero. As shown in FIG. 7, thelog data includes the frame identification number, the time stamp, andthe aggregated number of objects, but does not include the total numberof objects for each object and the metadata. In other words, in a casewhere the aggregated number of objects is zero, the record of the logdata does not have the data area.

As a comparative example, a case where both of the header area and thedata area of the log data is set to a fixed length is considered. Inthis case, when the aggregated number of objects is zero, the totalnumber of objects for each object is zero. In the data area, a defaultvalue (for example, zero) is arranged. With the data structure of thelog data according to the embodiment, it is possible to omit uselessinformation compared to the comparative example, and to suppress anincrease in data size of the log data.

FIG. 8 shows an example of log data generated by the informationprocessing device 12. Fields A to C of the log data of FIG. 8 are theframe identification information, the time stamp, and the aggregatednumber of objects, respectively. Each of fields D to I is the number ofdetections of each object classification, and here, an example where thenumber of types of objects to be detected is six is shown. Specifically,the fields D to I are the number of detections of automobiles, thenumber of detections of pedestrians and bicycles, the number ofdetections of motorcycles, the number of detections of constructionsites, and the number of detections of falling objects, and the numberof detections of traffic signals. Fields J to L are the tag information,and are the angle information, the pedestrian and the like crossinginformation, and the bright information, respectively. Fields M to X aremetadata related to a first object detected in each image frame.Specifically, the fields M to X are the detection frame coordinate (x),the detection frame coordinate (y), the detection frame width, thedetection frame height, the object classification, the absoluteposition, the relative position, the distance, the detection score, thedetection count, the degree of risk, and the signal color. A field Y andsubsequent fields are metadata related to second and subsequent objects.

As shown in FIG. 8, in first to fourth and sixth records, since theaggregated number of objects is equal to or greater than one, the totalnumber of objects for each object classification and metadata related toeach object are included. In a fifth record, since the aggregated numberof objects is zero, the total number (the number of detections) ofobjects for each object classification and metadata are not included. Inother words, among the records of FIG. 8, the record in which theaggregated number of objects is zero has data solely in the fields A toC, and does not have data in the field D and subsequent fields. In thefourth record, the aggregated number of objects is one. For this reason,the fourth record has solely one piece of metadata, and has data solelyin the fields A to X. In this way, it is possible to suppress anincrease in data size of the log data.

Generation Operation of Log Data

A generation operation of log data will be described referring to aflowchart of FIG. 9. In FIG. 9, an operation to generate log data basedon one image frame is shown. The operation is performed on each of aplurality of image frames.

Step S100: The controller 126 of the information processing device 12detects objects in each image frame through the image recognitionprocessing on the image frames.

Step S200: The controller 126 generates a record including theidentification information of each the image frame and the aggregatednumber of objects detected in the corresponding image frame.

Step S300: The controller 126 determines whether or not the aggregatednumber of objects is equal to or greater than one. In a case where theaggregated number of objects is equal to or greater than one, theprocess progresses to Step S400. In a case where the aggregated numberof objects is not equal to or greater than one, Step S400 is skipped,and the process ends.

Step S400: The controller 126 adds the total number of objects for eachobject classification to the corresponding record.

Step S500: The controller 126 adds the metadata related to each objectto the corresponding record. Then, the process ends.

As described above, in the information processing system 1 according tothe embodiment, each record of the log data includes information of theaggregated number of objects detected in each image frame. In a casewhere the aggregated number of objects is equal to or greater than one,the total number of objects (each number of detections) for each objectclassification and the metadata related to each object are added to thelog data. In a case where the aggregated number of objects is zero, eachnumber of detections and the metadata are not added to the log data. Ina case where the aggregated number of objects is zero, the total numberof objects for each object is zero. In this case, since there is nodetected object, the metadata does not need to be added. With theinformation processing system 1 according to the embodiment, in a casewhere the aggregated number of objects is zero, unneeded data is notadded to the record, and it is possible to suppress an increase in datasize of the log data. An increase in data size of the log data to besearched is suppressed, whereby a search speed is improved, and it ispossible to improve an information search technique of the log data.

In the information processing system 1 according to the embodiment, eachrecord of the log data includes information of each number ofdetections, and the sequence of the metadata in the data area of eachrecord is determined based on each number of detections. With this, itis possible to set the data area of the log data to a variable length,and to suppress an increase in data size of the log data. With the useof information of each number of detections, it is possible to specify aportion of the data area of the log data where needed metadata ispresent. Accordingly, with information of each number of detections, forexample, it is possible to limit the data area to be scanned, and toimprove an information search technique of the log data.

Although the disclosure has been described based on the drawings and theexample, it should be noted that those skilled in the art can easilymake various modifications and corrections based on the disclosure.Accordingly, it should be noted that such modifications and correctionsare included in the scope of the disclosure. For example, the functionsand the like included in respective means or respective steps can berearranged as long as there is no logical contradiction, and two or moremeans or steps may be combined into single means or step or may bedivided.

A configuration in which, for example, a general-purpose informationprocessing device, such as a smartphone or a computer, is made tofunction as the communication device 11, the information processingdevice 12, the imaging device 13, or the server 20 according to theabove-described embodiment may be made. Specifically, a program, inwhich processing contents for implementing the function of thecommunication device 11 and the like according to the embodiment aredescribed, is stored in a memory of the information processing device,and a processor of the information processing device is made to read andexecute the program. Accordingly, the disclosure according to theembodiment can also be implemented as a program to be executable by theprocessor.

For example, in the above-described embodiment, a part of processingoperations to be performed in the vehicle 10 may be performed in theserver 20, and a part of processing operations to be performed in theserver 20 may be performed in the vehicle 10. Specifically, for example,a configuration in which an image processing operation is performed bythe server 20 may be made.

In the above-described embodiment, although an example where themetadata in the log data is arranged following the sequence of thenumber of detections (a) to the number of detections (z) of the recordhas been described, the sequence may be replaced. For example, theserver 20 may change the sequence of the metadata in the log data in thepredetermined period based on at least one of the recognition frequencyand the detection score of each object in the predetermined period.Information of the sequence may be recorded in the sequence table suchthat the sequence of the metadata can be specified at the time of searchprocessing.

FIG. 10 shows an example of a sequence table. The sequence table isstored in the server storage unit 22 of the server 20. As shown in FIG.10, the sequence table includes information related to a range of aframe identification number and an arrangement sequence of objects. Inthe records included in the range of the frame identification number,the metadata is arranged along information related to the arrangementsequence of the corresponding objects. For example, in the recordrelated to the image frame having the frame identification number of00000001, the metadata is arranged along a sequence of No. 1 of thesequence table. For example, in the record related to the image framehaving the frame identification number of 00100001, the metadata isarranged along a sequence of No. 2 of the sequence table. In thesequence of No. 2 of the sequence table, the sequences of c and d arereplaced compared to the sequence of No. 1. In a sequence of No. 3 ofthe sequence table, the sequences of a and b are replaced compared tothe sequence of No. 2. In this way, the sequence of metadata is changedbased on at least one of the recognition frequency and the detectionscore in the predetermined period, whereby it is possible to improve asearch speed.

The tag information included in the log data may be optimized. Forexample, the server controller 23 of the server 20 measures a searchspeed of the log data with a predetermined search query based on the taginformation, and in a case where the search speed is lower than apredetermined value, may perform change by merging or subdividing thetag information. The server controller 23 of the server 20 changes thetag information such that the search speed becomes equal to or higherthan the predetermined value. In this way, the search speed of the logdata is evaluated using the predetermined search query and the taginformation is changed, thereby improving a search speed of the logdata.

In the above-described embodiment, although an example where all objectsto be detected are detected through the image recognition processing hasbeen described, the disclosure is not limited thereto. For example, theinformation processing device 12 may omit detection of a part of objectsbased on a result of the image recognition processing. For example,specifically, at least a part of object classifications to be detectedmay be hierarchized, and solely in a case where a detection score ofimage recognition processing related to a higher hierarchical objectclassification is equal to or greater than a predetermined value, imagerecognition processing related to a lower hierarchical objectclassification of the object classifications may be executed. FIG. 11 isa conceptual diagram representing the hierarchical relationship betweenobject classifications to be detected. An object classification X is ahigher hierarchical object classification (for example, vehicle), andobject classifications A to C (for example, heavy vehicle, mediumvehicle, and compact vehicle) are included directly below the objectclassification X. Object classifications a, a′ (for example, truck andbus) are included in a lower hierarchy of the object classification A,and object classifications b, b′ are included in a lower hierarchy ofthe object classification B. For example, in a case where a detectionscore related to the object classification A is equal to or greater thana predetermined value, detection related to the object classificationsa, a′ in the lower hierarchy of the object classification A isperformed. In a case where a detection score related to the objectclassification A is less than the predetermined value, detection relatedto the object classifications a, a′ in the lower hierarchy of the objectclassification A is not performed. In this way, it is possible to omitdetection of an object classification for which a detection score islow, and the metadata related to the object classification is notincluded in the log data. For this reason, it is possible to suppress anincrease in data size of the log data, and to increase a speed of searchof the log data. In regard to an object classification for which adetection score is high, since image recognition processing of lowerhierarchical subdivided object classifications is executed, and themetadata related to the object classifications is included in the logdata, it is possible to perform more subdivided search.

What is claimed is:
 1. A method of generating log data that includes aplurality of records, in which information detected through imagerecognition processing on a plurality of image frames is recorded, themethod comprising: detecting objects in each image frame through theimage recognition processing; generating, for each image frame, a recordincluding identification information of a corresponding image frame andthe aggregated number of objects detected in the corresponding imageframe; and adding metadata related to the detected objects to acorresponding record when the aggregated number of objects is equal toor greater than one, wherein when the aggregated number of objects iszero, the metadata related to the detected objects is not added to therecord.
 2. The method according to claim 1, further comprising addingthe total number of objects for each object classification to thecorresponding record when the aggregated number of objects is equal toor greater than one.
 3. The method according to claim 1, wherein asequence of metadata in the log data is determined based on at least oneof a recognition frequency and a detection score of each object.
 4. Themethod according to claim 1, further comprising: changing a sequence ofmetadata in the log data in a predetermined period based on at least oneof a recognition frequency and a detection score of each object in thepredetermined period; and recording information of the sequence in asequence table.
 5. The method according to claim 1, further comprising:measuring a search speed of the log data with a predetermined searchquery based on tag information included in the log data; and changingthe tag information when the search speed is lower than a predeterminedvalue.
 6. The method according to claim 1, wherein: at least a part ofobject classifications is hierarchized; and solely when a detectionscore of image recognition processing related to a higher hierarchicalobject classification is equal to or greater than a predetermined value,image recognition processing related to a lower hierarchical objectclassification of the higher hierarchical object classifications isexecuted.
 7. A non-transitory computer-readable medium storing a programthat causes a computer functioning as an information processing deviceconfigured to generate log data including a plurality of records, inwhich information detected through image recognition processing on aplurality of image frames is recorded, to execute processing, theprocessing comprising: detecting objects in each image frame through theimage recognition processing; generating, for each image frame, a recordincluding identification information of a corresponding image frame andthe aggregated number of objects detected in the corresponding imageframe; and adding metadata related to the detected objects to acorresponding record when the aggregated number of objects is equal toor greater than one, wherein when the aggregated number of objects iszero, the metadata related to the detected objects is not added to therecord.
 8. A data structure of log data including information detectedthrough image recognition processing on a plurality of image frames, thedata structure comprising a plurality of records corresponding to theimage frame, wherein: each record includes identification information ofa corresponding image frame and the aggregated number of objectsdetected in the corresponding image frame; and a record in which theaggregated number of objects is equal to or greater than one includesmetadata related to the detected objects, and a record in which theaggregated number of objects is zero includes no metadata.
 9. A datastructure of log data that is used in an information processing systemincluding a vehicle and a server, and includes information detectedthrough image recognition processing on a plurality of image frames, thedata structure comprising a plurality of records corresponding to theimage frame, wherein: each record includes identification information ofa corresponding image frame and the aggregated number of objectsdetected in the corresponding image frame; and the vehicle adds metadatarelated to the detected objects to the corresponding record when theaggregated number of objects is equal to or greater than one and adds nometadata when the aggregated number of objects is zero.